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Empirical  Evaluation  of  the  Virtual  Autonomous  Navigation  Environment 

ABSTRACT 

The  US  Army  Corps  of  Engineers'  (USACE)  Virtual  Autonomous  Navigation  Environment  (VANE)  is  a  physics-based,  multi-scale 
numerical  testbed  designed  to  quantitatively  and  accurately  predict  sensor  and  autonomous  system  performance  in  a  simulation 
environment.  The  work  presented  here  captures  progress  on  an  initial  empirical  evaluation  of  how  well  the  current  VANE  system  is  able  to 
reproduce  a  real  autonomy  system’s  perception  performance.  Findings  will  directly  guide  continuing  development  of  VANE,  while 
beginning  to  develop  a  suite  of  example  sensor  models  and  virtual  environments. 

This  first  experiment  focuses  on  testing  world  modeling  and  sensor  simulation.  Data  was  collected  from  the  Crusher  autonomous  vehicle, 
developed  under  the  DARPA  UPI  program.  Some  sensor  data  was  collected  and  manually  processed  to  produce  a  VANE  scene  model. 
Crusher  was  again  driven  through  the  real  scene  to  collect  real  sensor  data  as  the  baseline  sensor  data.  The  positions  of  the  sensors  were 
extracted  and  was  used  to  generate  a  VANE  simulation  to  exactly  match  Crusher's  path.  Both  datasets  were  fed  to  an  offline  version  of 
Crusher's  autonomous  perception  software.  The  outputs  from  the  two  separate  input  data  sets  were  compared.  The  results  indicate  good 
agreement  between  the  outputs,  especially  on  solid  ground  and  solid  objects.  Differences  were  observed  in  the  areas  of  vegetation,  an  area 
requiring  further  work  to  improve  modeling  and  simulation  of  the  sensors.  Greater  accuracy  will  also  be  required  in  the  ground  truth  data, 
which  was  collected  at  WAAS  GPS  quality  rather  than  RTK2  quality. 
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UGV  Autonomy  Systems 


NREC  Vehicle  “Crusher”,  performing  navigation 
tasks  for  Darpa  UPI  Field  Testing 
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Simulation  as  a  Test  Harness 
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VERSAT2  (NREC/TARDEC) 
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Main  Challenge:  Real-Time  Fidelity 


•  Simulators  are  useful,  but  lack  fidelity 

•  World 

-  Limited  polygons,  e.g.  no  grass  or  similar  (1  blade/ mm2  ->  106  blades/ m2) 

-  No  mud,  water,  or  similar... 

•  Ladar  Sensors 

-  No  motion  during  scan  (—13  msec  staticshots) 

-  No  range  or  angular  noise,  with  first  return  only 

-  No  attention/ non- returns  (reflectance,  range,  foreshortening,  ...) 


•  I  maging  and  Stereo  sensors 

-  Easy  to  add  but  difficult  to  model  well 


Vehicle 

-  No  suspension,  tire  ground  modeling,  friction/dynamics  modeling 
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Example  from  Real  Data 


Lack  of  fidelity  significant  impacts  simulation  vs 
real  vehicle  performance  comparisons 
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VANE/NREC  Effort 


•  Summary  so  far: 

-  Simulation  is  very  useful  for  UGVs,  but  fidelity 
gap  limits  its  use  and  effectiveness 

•  However,  ERDC's  VANE  is  a  high  fidelity 
simulator  derived  from  physics  models 

•  VANE/NREC  Project  goal 


I  nvestigate  if  VANE  can  address  fidelity  gap  and 
create  realistic  simulation  environment 
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Evaluating  VANE 


•  How  do  we  evaluate  simulator  quality? 

—  Run  robot  autonomy  and  evaluate  resulting  decisions 

•  Should  match  real  vehicle  decisions  in  the  real  world 

•  Path  planning  cost  maps  represent  this  knowledge 


•  NREC/ERDC  approach 

—  Simulation  of  a  known  world  location  and  collect  real 
data  from  that  location 


-  Compare  cost  maps  generated  by  robot  autonomy 
perception  on  simulated  sensory  data  and  real  data 
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UGV's  and  simulation 
Problem  approach 
Real  robot  data 
Simulated  results 
Cost  map  Comparisons 
Conclusions 
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Approach:  Data  Collection  (NREC) 


1.  Collect  field  test  data  from  a  real  UGV 

•  Data  stored  to  time-stamped  logs 


6  Tilting  LiDars 
4  Camera  “Cubes” 

IxRGB  Stereo  head 
IxNIR  Camera 
IxDark  Red  Camera 
DGPS/RTK  INS  system 
Suspension  sensors 
Many  vehicle  sensors 
Calibrated  models 
NREC  Autonomy  SW 


Developed  on  the  UPI,  UGCV,  and 
PerceptOR  programs 
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Approach:  Model  Generation 


2.  Extract  world  model  data  and  generate 
VANE  model  of  a  real  environment 


Manual  photos,  world  description, 
physical  plant  specimens 


UPI  Log 
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Vehicle  imagery 


Vehicle  pose,  sensor  poses 


ERDC  VANE 
model  generation 
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Approach:  Simulation  (ERDC) 


3.  Run  VANE  Simulator  and  generate 
Simulated  sensor  data 

•  Data  registered  to  true  vehicle  sensor  poses 


Approach:  Run  Autonomy  (NREC) 


4.  Run  NREC  autonomy  SW  on  simulated 
and  real  data  and  compare  cost  maps 
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UPI  Software 
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•  UGV's  and  simulation 

•  Problem  approach 

•  Real  robot  data 

•  Simulated  results 

•  Cost  map  Comparisons 

•  Conclusions 
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Testing  Site 


Fort  Drum  NY,  J  une  2008,  with  an  open  field,  rocks,  small  vegetation 
and  hay  bails 

-  Positive  obstacles  (rocks,  hay  bails) 

-  “Soft"  obstacles  (vegetation) 


Navigable, 
Low  cost 


'  Obstacles, 
High  cost 


Free  space, 
Very  low  cost 
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Data  Collection 


•  6  Runs  recorded  at  around  6pm  with 
NREC's  Crusher  vehicle 

•  Only  forward  looking  sensors  used 

•  5  Runs  used  for  model  construction 

-  Data  provided  to  ERDC 

•  1  run  held  back  for  evaluation 

-  Sensor  poses  provided  to  ERDC  for  simulation 
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NATlofOTroMB 

SICK  LMS  291 -SI 4  sensor.  Scans  a  line  with  equal  angle  spqgjflcjl(Ei$gh  scan 
range  readings  left  to  right  in  0.5°  deg  increments  [-45°,  TtfE^fOS'^rtiOS  NSpiiPJFTSr  awSSlRlNG  CEN 


Tilting  Ladar  Sensors 


Tilting  -45° 

variable  tilt  rate  (upx,  upy,  upz) 


**  All  coords  in  UTM 
X  =  Easting  (m) 

Y  =  Northing  (m) 

Z  =  Height  (m) 


Pose:  (x,  y,  z) 


Sick  Scan  Model 


View  from  above 


Looking  direction 


-0.5° 

spacing 


-45° 

Returned 

distance 


181  readings  per  scan. 
Scans  at  75Hz 


45.5° 


23.2  15.1  29.6 


Range  values  returned 


Camera  Sensors 


All  coords  in  UTM 


(upx,  upy,  upz) 

2  x  RGB  Cameras  Pose:  (*.  Y,  z) 

1  x  Red 


X  =  Easting  (m) 
Y  =  Northing  (m) 
Z  =  Height  (m) 


2 

1  x  Red 
1  x  Near  IR 


Kb  I 


Cameras  based  off  the  Bumblebee  1  stereo  head  from  PointGrey.  The  color  stereo  hea fj t.L  RbBCTICS 


Perspective  Camera  Model 


Optical  center 
(corresponds  to 
“lens”  location, 
and  “eye”  point) 


up 


Looking  direction 
Aligned  with  optical  axis 


Ideal  pinhole  model.  Defined  by 
Effective  focal  length  f x  ~  f y  =  f 
Optical  center  assumed  to  be  in  center  of  image 
Imaging  surface  size  (w,  h) 
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Real  Sensor  Data:  Imagery 


•  Front  looking  right  cameras 


Left 


Perspective 
affects,  light 
scattering, 
diffuse  lighting, 


Red 
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Real  Data:  Colorized  Ladar  Scans 


Reconstructed 
from  vehicle 
poses 
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General  Data  Flow 


•  Sensor  data  for  model  and  sensor  poses  for 
simulation  sent  to  ERDC 

NREC  ERDC 


Pt  cloud,  imagery 


Sensor  poses 
Intrinsics 


Binary  sensory  data: 
Lidar  scans,  RGB- 
Red/NIR  imagery. 


Results 
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ERDC  Generated  Scene 


ERDC  Generated  Scene 
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Cost  Map  Comparisons 

•  Run  both  sets  of  data  through  UPI  System 
and  produce  cost  maps 

•  Cost  map  representation: 

-  Low  cost  represents  "free  space" 

-  High  cost  represents  "obstacles" 

-  Vegetation  often  in  between 
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Cost  Map  Analysis 


•  Visualizations: 

-  Log  of  cost  value  (to  show  dynamic  range) 

•  Evaluations 

-  Direct  pixel  subtraction 

—  Subtraction  with  median  filtering  (reduce  edge 
effects  that  may  occur) 
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Real  Data  Cost  Map 


1 

1  ,r  mMwi  '  3? 

March  2009 


Higher  values  indicate  higher  cost  areas  ie: 
areas  vehicle  prefers  not  to  go  through. 
Log(cost)  is  shown  to  show  dynamic  range 
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Real  Data  Cost  Map 


Major  obstacles 


Vegetation 
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Areas  of  high  cost  due  to  vegetation 
and  occluded  ground  layer 
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Higher  values  indicate  higher  cost  areas  ie: 
areas  vehicle  prefers  not  to  go  through. 
Log(cost)  is  shown  to  show  dynamic  range 
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Simulated  Cost  Map 


Log  Sim  Cost  Map 
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Higher  values  indicate  higher  cost  areas  ie: 
areas  vehicle  prefers  not  to  go  through. 
Log(cost)  is  shown  to  show  dynamic  range 
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Simulated  Cost  Map 


Hay  Bails 


Vegetation 


No  area  of  high  cost 
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Higher  values  indicate  higher  cost  areas  ie: 
areas  vehicle  prefers  not  to  go  through. 
Log(cost)  is  shown  to  show  dynamic  range 
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Raw  absolute  Pixel  Subtraction 


Median  Filtered  Difference 


Median  Filtered  Difference  (9x9) 
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Discussion 


•  Coarse  comparison  is  good 

-  Strong  expectation  that  vehicle  will  follow  same  trajectories 

-  Geometric  obstacles  (hay  bails)  produce  very  similar  costs 

•  Key  cost  differences  for  area  just  behind  vegetation 

•  Deeper  analysis  shows  higher  cost  is  associated  with 
ground  estimation  in  real  data 

-  If  ground  level  is  inferred  by  vehicle  (vs.  directly  being  observed), 
cost  estimate  is  higher 


Conclusions 


•  First  phase  comparisons  are  good 

—  Strong  evidence  that  vehicle  trajectories  in  simulation 
will  match  real  vehicle  performance 


•  Some  challenges  identified 


-  Pose  error  in  data  collection  is  an  issue 

-  No  good  tools  for  rapid  model  building 

-  Vegetation  differences  (for  perception)  may  become  a 
more  significant  issue  in  more  complex  terrain 
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